Type a few words into a search engine and the answer comes back in a fraction of a second, drawn from an index of many billions of pages. Your bank flags a suspicious payment while you are still standing at the till. A streaming service updates "what to watch next" for hundreds of millions of people overnight. None of that happens on one computer. None of it could.
Data engineering begins at the moment one machine stops being enough: not enough disk, not enough memory, or just not enough time. The moment you spread a job over many machines, three new problems appear that did not exist before. Machines fail all the time, so the system must work with some of them gone. Data has to be divided in a way that lets the machines work independently. And there is no single clock, so "what happened first?" needs an answer. A graduate keeps these three problems in mind and knows the standard answers.
A fair warning that experienced engineers repeat: many datasets labelled "big" fit comfortably on one modern server, and a single well-written program on one machine often beats a cluster. Frank McSherry, Michael Isard and Derek Murray made the point in a 2015 paper, "Scalability! But at what COST?", by showing a laptop outrunning several distributed systems on some graph tasks. The tools below are worth it only when the data or the speed really demands them.
Start with the cheapest calculation in the subject. A hard disk can read sequentially at very roughly 150 megabytes a second, a SATA solid-state drive at about 500, and a fast NVMe drive at a few gigabytes a second. These are round figures that vary by model; the pattern matters, not the digits. A terabyte is a million megabytes, so one hard disk needs about 6,700 seconds, nearly two hours, to read it once.
That is the whole reason for the field. The catch is what else scales with the number of machines: failures. In 2024 Backblaze reported an annualised failure rate of 1.57% across about 299,000 of its drives. For one drive that is rare. For 10,000 drives it is about 157 failures a year, roughly three a week, so at that size a failure is routine, not an emergency. Systems built for scale assume parts will break and carry on. Try the arithmetic.
Illustrative round speeds, perfect even splitting and no network limits, which real jobs never have. Failure figure: Backblaze Drive Stats for 2024 (annualised failure rate 1.57%, about 299,000 drives).
Without JavaScript: 1 TB on one hard disk at 150 MB/s takes about 1.9 hours; across 100 disks, about 67 seconds. 1,000 TB across 1,000 disks takes about 1.9 hours again.
To use many machines you must decide where each piece of data lives. The simplest answer is a hash: run the key (a user ID) through a scrambling function and take the remainder when dividing by the number of machines. Every key lands on a predictable machine, and the load spreads evenly. It has one terrible flaw. Add a machine and the divisor changes, so almost every key now maps somewhere else, and nearly all your data must move. Going from 3 machines to 4 moves 3 keys in 4.
Consistent hashing, introduced by David Karger and colleagues in 1997 for web caches, avoids this. Put the machines on a circle by hashing their names, and give each key to the first machine clockwise from its own hash. A new machine takes over only a slice of the circle, so about one key in N + 1 moves. Using many points per machine ("virtual nodes") evens out the load. Amazon's Dynamo (2007) and many databases built since use the idea.
10,000 keys named user-0 to user-9999, scrambled with a 32-bit FNV-1a hash plus a finishing mix; the circle uses 100 points per machine. Both columns use the same keys.
Without JavaScript: going from 3 machines to 4, remainder hashing moves 75% of the keys and consistent hashing about 26%. From 9 to 10: about 90% against 10.5%.
Once data is spread across machines, how do you compute over it? In 2004 Jeffrey Dean and Sanjay Ghemawat of Google published MapReduce, a programming model that hid the hard parts. You write two small functions. Map looks at one record and emits key-value pairs. The system then shuffles: it groups all pairs with the same key onto one machine. Reduce combines the values for each key. The framework handles splitting, retrying failed tasks and moving data. Doug Cutting and Mike Cafarella built the open-source Hadoop from the idea around 2006, with Yahoo as an early backer.
Counting words is the standard example. Each mapper emits ("the", 1) for every "the" it sees. The shuffle sends all the "the" pairs to the same reducer. The reducer adds them. Nothing in your two functions mentions machines, so the same code runs on 3 lines or 3 billion. Its weakness is that results between stages were typically written to disk; Apache Spark (Zaharia and colleagues, NSDI 2012) kept working data in memory and became the more common choice for multi-step jobs. The shuffle is the expensive step, because it moves data across the network. A combiner adds up counts inside each mapper first, so fewer pairs travel.
Six invented lines are dealt to the mappers in turn. Words are sent to a reducer by hashing. Every count shown is computed by the page.
Without JavaScript: six short lines of text contain 34 words. Without a combiner, all 34 pairs cross the network in the shuffle; with a combiner on 2 mappers, 18 do.
With that failure rate, you cannot keep a single copy of anything. The standard answer is replication: keep N copies on different machines (three is a common default in Hadoop's file system). Lose one and two remain. The new problem is that copies can disagree. A write reaches some replicas first; a read that lands on a stale one returns old data.
Amazon's Dynamo paper made the trade-off explicit with three numbers. N is the number of replicas, W the number that must confirm a write, R the number asked on a read. If R + W is greater than N, any set of R replicas must overlap any set of W replicas, so a read always meets at least one up-to-date copy. If not, a read can miss the latest write. Bigger W and R mean stronger guarantees and slower, less available operations, because you are waiting on more machines. This is the practical content of the CAP theorem (Eric Brewer's 2000 conjecture, proved by Seth Gilbert and Nancy Lynch in 2002): when the network splits, you choose between answering and being sure the answer is current. Martin Kleppmann has argued that the usual "pick two of three" summary is misleading, since partitions are not optional.
Model: a write lands on W random replicas and the rest lag; a read asks R random replicas and sees the latest write if any of them has it. Exact value from counting, plus a 20,000-trial simulation.
Without JavaScript: with N = 3, W = 1 and R = 1, a read is stale two times in three (66.7%). With W = 2 and R = 2, R + W = 4 is greater than 3 and reads are never stale. With N = 5, W = 2, R = 2 the stale chance is 30%.
Analytical questions usually touch a few columns of a wide table: total sales by country uses two of the twenty columns. Storing a table row by row, as most transactional databases do, forces you to read every column just to reach the two you want. Columnar formats store each column together. Queries read only the columns they need, and a column of similar values compresses far better than a row of mixed ones. The idea was used in research systems in the 2000s and spread through Google's Dremel paper (2010) and open formats such as Apache Parquet (2013).
Assumes 20 columns of 8 bytes each and a scan speed of 1 GB per second across the cluster (illustrative). Real compression ratios depend on the data.
Without JavaScript: for 1 billion rows of 20 columns at 8 bytes each (160 GB), a row layout reads all 160 GB, about 160 seconds at 1 GB/s. A query that needs 2 columns reads 16 GB in a column layout, 4 GB if the columns compress 4 times.
Many pipelines process streams: clicks, sensor readings, payments. Computing "events per minute" sounds trivial until you ask whose minute. Event time is when the thing actually happened, stamped on the phone or the sensor. Processing time is when your system saw it. Between the two sit slow mobile networks, retries and a phone in a tunnel, so events arrive late and out of order. Count by arrival and a burst of delayed events lands in the wrong minute.
The remedy, popularised by Google's Dataflow model (Akidau and colleagues, 2015) and open-source engines such as Apache Flink and Beam, is to window by event time and track a watermark: the system's estimate of "I have probably seen everything up to time T". Wait longer after a window ends and you drop fewer late events but answer later. There is no free setting. Logs such as Apache Kafka, created at LinkedIn and open-sourced in 2011, store the events in order so a failed job can replay them. Replaying only works safely if each step is idempotent: running it twice gives the same result as once, the last interactive below.
40 invented events in the first minute, each delayed by a random amount up to the slider. Windows are 10 seconds. Windows beyond 60 seconds only receive late arrivals.
Without JavaScript: with delays up to 20 seconds, counting by arrival puts only 1 event in the first 10-second window where 7 really happened. Waiting 10 seconds after each window closes drops 9 of the 40 events as too late; waiting 20 drops none.
Distributed jobs fail halfway, and the system's response is to run them again. That is only safe if running a step twice has the same effect as once. A job that appends its five rows to a table will, after a crash at row 3 and a retry, leave eight. A job that replaces the whole day's partition, or inserts each row by its unique key and updates it if it exists, leaves five whichever way it fails. Real systems add other tools (transactions, atomic file renames, deduplication keys), and vendors often advertise "exactly-once" delivery, which in practice means at-least-once delivery plus idempotent writes. Try it.
Without JavaScript: "append" leaves 8 rows after a crash and retry; "replace the day" and "upsert by key" leave 5.
Scale buys speed by splitting data across machines, and charges for it in failures, copies, network traffic and confusion about time. The craft is knowing the standard answers: partition so that work divides, replicate so that failure is survivable, aggregate near the data, window by the time that matters, and make every step safe to repeat.
Try this: next time an app shows a count or a "last updated" time that seems slightly off, guess which of the ideas above is behind it: a lagging replica, a late event, or a retried job. Then check whether the number corrects itself later.
Sources. Backblaze, "Drive Stats for 2024" (annualised failure rate 1.57%; 298,954 drives meeting the inclusion criteria); McSherry, Isard and Murray, "Scalability! But at what COST?", HotOS 2015; Karger et al., "Consistent hashing and random trees," STOC 1997; Ghemawat, Gobioff and Leung, "The Google File System," SOSP 2003; Dean and Ghemawat, "MapReduce," OSDI 2004 (and Communications of the ACM 2008); Zaharia et al., "Resilient distributed datasets," NSDI 2012; DeCandia et al., "Dynamo," SOSP 2007; Brewer, PODC 2000 keynote; Gilbert and Lynch, ACM SIGACT News 2002; Kleppmann, "A critique of the CAP theorem," 2015; Melnik et al., "Dremel," VLDB 2010; Akidau et al., "The Dataflow Model," VLDB 2015; Apache Hadoop, Parquet and Kafka project histories. Disk speeds are round illustrative figures. All simulation outputs and counts were computed in node.